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Abstract 

Design-based research (DBR) is a core methodology of the Learning Sciences. 
Historically rooted as a movement away from the methods of experimental psychology, it 
is a means to develop “humble” theory that takes into account numerous contextual 
effects for understanding how and why a design supported learning. DBR involves 
iterative refinement of both designs for learning and theory; this process is illustrated with 
retrospective analysis of six DBR cycles. Calls for educational research to parallel 
medical research has led learning scientists to strive for more specific standards about 
what constitutes DBR and what makes it desirable, especially regarding robustness and 
rigor. A recent trend in DBR involves efforts to extend the reach through scalability. These 
developments potentially endanger the designerly nature of DBR by orienting focus 
toward generalizability, meaning researchers must be vigilant in their pursuit of 
understanding how and why learning occurs in complex contexts. 
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1. Overview of Design-Based Research 

Design-Based Research (DBR) is a core methodology of the learning sciences. Begun as a 
movement away from experimental psychology, DBR was proposed as means to study learning amidst the 
“blooming, buzzing confusion” of classrooms (Brown, 1992, p. 141). It is a way to develop theory that takes 
into account numerous contextual effects for understanding how and why a design supports learning; these 
theories are “humble in that they target domain-specific learning processes” (Cobb, Confrey, diSessa, 
Lehrer, & Schauble, 2003, p. 9). DBR involves iterative refinement of both designs for learning and theory 
(Brown, 1992; Collins, 1992; The Design-Based Research Collective, 2003). 

This paper outlines the methodological standards for conducting DBR, illustrated with an example, 
and describes recent advances. 


2. Methodological Standards for Conducting Design-Based Research 

DBR is sometimes conflated with mixed methods or action research; this, paired with calls for 
educational research to parallel medical research has led learning scientists to strive for more specific 
standards about what constitutes DBR and what makes it desirable, especially regarding robustness and 
rigor. This section details current methodological standards for conducting DBR. 

2.1. A collaborative effort conducted in context 

DBR is typically conducted as a team of researchers, designers, and practitioners with intensive 
planning and debriefing sessions throughout the process. Rather than a wholly researcher-driven process, 
practitioners generally have greater ownership of the process. Working collaboratively, they identify a 
practical problem (Reeves, 2006) that is then investigated through literature review, learning theory, and 
question posing. The intervention instantiates this learning theory into the design. 

Because learning is understood to be a process, and because DBR seeks understanding of how 
learning occurs, process data are prioritized in DBR, such as video records and artifacts of student work. 
This approach allows researchers to be opportunistic when something surprising or emergent occurs. The 
notion that emergence plays a central role in DBR is a shift away from the more positivistic origins in which 
variables are well-known a priori (Collins, 1992). 

DBR allows for intervention while yet valuing the importance of social interaction rather than social 
isolation (Collins, Joseph, & Bielaczyc, 2004). This resonates the basic belief by learning scientists that 
learning is a fundamentally social, interactional process. By occurring in classrooms rather than in 
laboratories, DBR also allows for testing of designs and theory that address “the complexity that is a 
hallmark of educational settings” (Cobb et al., 2003, p. 9). The challenge is to apply lessons learned in 
context to a broader range of settings (Barab & Squire, 2004). Because DBR may not be replicated in the 
classical sense, given strong ties to context, it is critical to share the design along with thick description 
(Barab & Squire, 2004). 

2.2. Iterative cycles refine the design and the theory 

Because of the contextual nature of DBR, some view DBR as a means to generate, but not validate 
conjectures about learning (Sandoval, 2004); however, because such conjectures are made visible in designs 
for learning, they become testable through iterative refinement. Simply conducting one study in the field 
does not qualify, although it may be reported as one cycle in a longer DBR effort. Iterative refinement across 
contexts allows conjectures to become robust (diSessa & Cobb, 2004) by placing theory “in harm’s way” 
(Cobb et al., 2003, p. 10). 

The development of Interactive Learning Assessments (ILAs) illustrates the iterative refinement 
process (McKay, Cantarero, Svihla, Yakes Jimenez, & Castillo, 2014; Phillips et al., 2009; Svihla et al., 
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2010; Svihla, Phillips, et al„ 2009; Svihla, Vye, et al„ 2009; Svihla et al„ 2013; Yakes et al„ 2013). ILAs 
place the learner in an authentic, professional role giving advice to virtual clients. 

ILAs were first developed in response to a call for high school biology assessments that did not 
pause learning, but instead assessed students as they learned; more specifically, we aimed to assess how 
students used resources to solve problems that were new to them. Because this call came from an 
organization interested in using our designs for all schools in one state, we faced early challenges; our design 
decisions were driven by the need for scalability. This led us to seek school partners to test our designs, but 
meant that we neglected some of the contextual influences that are typical of DBR. Initially, we did not 
involve instructors in the design process extensively, but we did debrief with them to inform redesign. We 
partnered with subject matter experts (e.g., a genetic counselor or dietitian) who helped ensure the problems 
reflected authentic professional practices, as this was central to our humble theory. 

Our designs for and theory of learning evolved through six iterations (Figure 1 & Table 1). We 
initially provided authentic, real-world problems posed by virtual clients and access to resources as a way to 
support students to solve complex problems. Students took on the role of interns and gave counsel to virtual 
(avatar) clients. Our first design succeeded in supporting learning, but was too open-ended to be a useful 
assessment at scale. Beginning with iteration 2, we designed more specified sequences of questions and 
provided feedback from a virtual supervisor. We found the ILAs supported learning and provided useful data 
for assessment, but the student experience was too linear and scaffolded. 

We moved to a new setting—a university nutrition program seeking to provide students with ways to 
learn about professional practices prior to internships as a means to recruit and retain diverse students (Svihla 
et al., 2013). With this different motivation driving our work, we sought to bring instructors more centrally 
into the role of designers of cases. To offset the linear feel of the cases, we sought to support greater agency, 
providing opportunities to make choices among story-like branches. Instructors found it cumbersome to 
design such cases. Instead of distancing the instructors from the design process, we changed how we 
instantiated agency into the design, creating short story-like loops; students could explore as many or as few 
of the loops as they liked. In these versions, students learned content and professional practices, and they 
enjoyed the opportunity to explore further according to their level of interest. 



Figure 1. Refinement of humble theory of learning instantiated in Interactive Learning Assessments 


Retrospective analysis of DBR cycles provides an opportunity to “see order, pattern, and regularity” 
in messy, complex settings (diSessa & Cobb, 2004, p. 84) and supports the development of “useful, 
generalizable theories” (Edelson, 2002, p. 112). This analysis includes considering the conditions for success 
(Dede, 2004) and highlights the need to report failures (O'Neill, 2012). 

Retrospective analysis of the six iterations - across varied contexts (rural, urban; high school, 
university; biology, nutrition) - highlights areas where our theory is robust: students consistently learned by 
taking on real roles and solving challenges posed by virtual clients. This hinged on our ability to place 
students in roles they could understand; when the role was further from their experience, the addition of 
vignettes of the virtual supervisor explaining the role bridged this gap. The distance between student 
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experience and professional role also affected feedback given to students. For high school students, it was 
hard to design feedback that did not seem schoolish, lowering the authenticity. In contrast, the university 
students found the opportunity to see an expert answer and compare it to their own answers to be an 
authentic learning activity. 
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Table 1. Design-based iterations in the development of Interactive Learning Assessments 


Iteration 

Participants 
and setting 

Role taken by 
students and 
problem 

Implementation 

Main findings 

Sample 

design 

decisions 

1 

Biology 

As a conservation 

One case 

Students learned 

Increase 


students (n—34) 

geneticist, student 

completed as 

about genetics 

scaffolding, 


at a rural 

advises developers 

think aloud task 

and saw what 

create 


southern US 
high school 

on conservation of 
two bird species 

with researcher 

they were 
learning as 
relevant 

diagnostic 
yet authentic 
multiple 
choice 
questions 

2 

Biology 

As an intern genetic 

One case 

Students didn’t 

Provide 


students (n=24) 

counselor, student 

completed as 

understand what 

explicit 


at a rural 

counsels couple 

think aloud task 

an internship 

guidance 


southern US 
high school 

worried about 
potential for having 
a baby with sickle 
cell disease 

with researcher 

was, but did 
learn genetics 
content from the 

case 

about 

internship 

3 

Biology 

As an intern genetic 

One case 

Students who 

Add 


students (n=48) 

counselor, student 

completed in 

moved quickly 

generate 


at an affluent 
west coast US 
suburban high 
school 

counsels couple 
worried about 
potential for having 
a baby with sickle 
cell disease 

class session 

from reading the 
problem to 
searching for 
information 
struggled; 
teacher unsure 
how/when to use 

case 

ideas step 
and 

reflective 
prompts, 
make less 
linear; add 
teacher-as- 
designer 

4 

Undergraduate 

As an intern 

One case 

Students learned 

Replace 


nutrition 

dietitian, student 

completed as 

and retained 

branches 


students (n= 15) 
at a 

southwestern 

US research 
university 

counsels family 
about nutrition 
needs of child with 
Down syndrome 

an online 
assignment 

content; 
designing 
branches was 
burdensome for 
instructor 

with loops 

5 

Graduate 

As an intern 

One case 

Students learned 

Create more 


nutrition 
students (n=14) 
at a 

southwestern 

US research 
university 

dietitian, student 
counsels pregnant 
woman about 
gestational diabetes 

completed as 
an online 
assignment, 
one in-class 
discussion 
session 

and retained 
content; 
instructor could 
design and teach 
with the case 

cases 

6 

Undergraduate 

As an intern 

Seven cases 

Students learned 

Investigate 


nutrition 

dietitian, student 

completed in 

and retained 

ways to 


students (n=25) 

counsels a range of 

place of class 

content; 

make 


at a 

clients on various 

meetings, plus 

instructor 

branching 


southwestern 

US research 
university 

nutrition topics 

seven in-class 

discussion 

sessions 

developed more 
student-centered 
practice 

design 

feasible 
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3. Extensibility of DBR: Design-Based Implementation Research (DBIR) 

In the earlier example of ILAs, the initial goal was to help bring about statewide systemic change by 
providing a new way to embed assessment within learning. This driver necessitated changes to traditional 
DBR. When we changed settings, we also changed the role of the instructors from informants and consumers 
to designers of cases; this shift reflected our goal to help bring about smaller scale yet systematic change 
within a university program. In the first set of high school iterations, instructors were uncertain about how to 
use the cases. In the first iterations in the university setting in which the cases were designed by instructors, 
the cases were treated as homework, supplemental to in-class lectures. In the most recent iteration, the same 
instructors replaced lectures with the cases and further supplemented them with discussion (McKay et al., 
2014). What we first viewed as a better assessment tool evolved into a tool for instructors to test their ideas 
about learning, resulting in more learner-centered teaching. 


3.1. Design-Based Implementation Research 

Recently, others have similarly sought ways to expand the reach of DBR, such as through 
“implementation paths” that could lead the way to scaling a design (Bielaczyc, 2013), seeking to develop 
learning theory that can be adapted to contexts (Barab & Squire, 2004), and Design-Based Implementation 
Research (DBIR, Fishman, Penuel, Allen, Cheng, & Sabelli, 2013; Penuel & Fishman, 2012). DBIR includes 
“(a) a focus on persistent problems of practice from multiple stakeholders’ perspectives; (b) a commitment to 
iterative, collaborative design; (c) a concern with developing theory related to both classroom learning and 
implementation through systematic inquiry; and (d) a concern with developing capacity for sustaining 
change in systems” (Penuel, Fishman, Haugan Cheng, & Sabelli, 2011, p. 331). 

In one example of DBIR, researchers partnered with four districts to develop a theory of action 
around improving mathematics instruction (Cobb, Jackson, Smith, Sorum, & Henrick, 2013); the partnership 
lasted four years through cycles of data collection and analysis focused on the strategies as implemented. In 
each cycle, they documented the intended strategies, recorded how they were actually enacted, and made 
recommendations based on analysis. In order to support and maintain the relationship between researchers 
and practitioners, the team used two means of data collection and analysis: first, they prioritized providing 
usable evidence for the districts to evaluate the impact of their policies; second, they iteratively tested their 
theory of action to refine it. In addition to being guided by and refining a theory of action, they created an 
interpretive framework; this tool was used to evaluate and guide design decisions prior to, during and after 
implementation. Following the four cycles of implementation, they began retrospective analysis to further 
test and refine their theory of action. This example highlights many parallels with DBR, including 
collaborative and contextual work with a focus on refining design and theory through iterative refinement 
and retrospective analysis. It also highlights the different scale at which DBIR is conducted, involving many 
districts, schools, and classrooms, and a focus on creating sustainable change. By working at this scale, the 
research is more easily generalizable; by testing conjectures across four districts, they were able to learn 
about strategies that were effective across districts given specific conditions. Because the target of their 
design was tied to how districts could support improved mathematics instruction, they were able to identify 
and refine strategies that were ineffective. For instance, school leaders had been receiving content- 
independent professional development to guide their feedback to mathematics teachers; however, this 
process uncovered that they were not able to distinguish between high and low quality enactments of the 
mathematics. By recommending school leaders instead receive content-based professional development, they 
were able to design a sustainable, lasting change. 

DBIR researchers emphasize the practical nature of their work, from problem to design to theory 
(Dolle, Gomez, Russell, & Bryk, 2013). This approach takes a broader view of the context and attends to 
usability by jointly considering how to change larger entities or systems (e.g., school districts) and how to 
support their ability to sustainably adapt designs (Penuel & Fishman, 2012). DBIR has only begun to be 
taken up, bringing focus on scalability and sustainability, while respecting teachers and avoiding trying to 
“teacher-proof’ the materials of reform, for instance, through productive adaptation. 
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3.2. Productive Adaptation 

One approach to DBIR is in teachers’ productive adaptations of curricula; this means staying faithful 
to the original intent of the design, reproducing “invariant principles” across sites while being responsive to 
local contexts (Kirshner & Polman, 2013). In particular, focusing on maintaining or increasing - rather than 
reducing - the complexity and students’ engagement can support productive adaptations (DeBarger, 
Choppin, Beauvineau, & Moorthy, 2013). 

Dialogic interactions between teachers and researchers can support productive adaptions (Kirshner & 
Polman, 2013), but deliberate support - and spaces - are needed to ensure these are frequent enough and 
sustained (Donovan, Snow, & Daro, 2013). Related to this, it is also important to attend to power 
relationships and ownership of problems of practice; researchers bring different cultural norms and may have 
status not afforded to practitioners. Deliberately viewing this as a cultural exchange, in which researchers 
and practitioners can trade ideas, can mitigate these challenges (Penuel, Cobum, & Gallagher, 2013). 

In some cases, district support for any particular program, professional development, or curriculum 
may be taken as another in a sequence of top-down mandates, and therefore meet with resistance at school 
sites (Borko & Klingner, 2013). This highlights the importance of attending to influences across levels in the 
system in which research is occurring. Because of this systems approach, not all DBIR research occurs 
within schools or formal settings; though less common, DBIR research has been conducted in communities, 
as a means to identify issues that might prevent youth from being successful and address them in creative, 
cross-institutional ways (McLaughlin & London, 2013). Such approaches are important because DBR has 
been critiqued for not sufficiently attending to equity and social justice (Confrey, 2005), though some work 
has sought this out (e.g., Barab, Dodge, Thomas, Jackson, & Tuzun, 2007). 


4. Are DBR and DBIR Designerly? 

Although design-based, not all DBR and DBIR appear to be designerly (Cross, 2001), explicitly 
applying design process by seeking needs, optimizing the design, and evaluating a solution in light of 
identified needs (Edelson, 2002). Because the targets of DBR are designs for learning and theories of 
learning, potential needs may be found both in review of research and in the world. Needs are sometimes 
implicit and the design process left to the reader’s imagination (e.g., “the tool was designed to scaffold 
learning of argumentation”). Aiming at scalability can strip the contextualist, designerly aspects from DBR, 
but committing to novel usability —and therefore a focus on context— can mitigate this. DBIR focuses on 
design at scale, which would suggest a less designerly approach; yet, the emphasis on working in partnership 
with practitioners to support sustained change has helped focus DBIR research on worldly needs. 

As these methods continue to evolve and incorporate bigger systems and big data, there are many 
opportunities for looking across streams of related data, such as logfiles and videos. These offer ways to 
evaluate the influence and refinement of designs for learning and of learning theories that are contextual and 
adaptive to the systems in which they reside. 

4.1 Credibility of Design-Based Research 

Concerns have been raised previously about the credibility of educational research in general (Levin 
& O'Donnell, 1999; National Research Council, 2002), urging researchers to employ methodologies 
influenced by medical research. In such approaches, tests of efficacy (whether the treatment works under 
optimal conditions) and effectiveness (whether the treatment works under real world conditions) “are often 
conflated” (Sloane, 2008, p. 625). Influenced by this, discussions about DBR have focused on robustness, 
rigor and validity, grounded in experimental perspectives, an odd choice given the contextual, qualitative 
work that is commonly done with DBR. However, trustworthiness and credibility - as applied in qualitative 
methods - have also been considered (Barab & Squire, 2004), resulting in other ways to evaluate DBR: 
Methodological alignment means the “research methods we use actually test what we think they are testing” 
(Hoadley, 2004, p. 203). Edelson holds that DBR should not be evaluated by the same standards as 
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traditional approaches because the goals differ; instead, “novelty and usefulness” of the theory developed 
should be applied (2002, p. 118). 


4.2 New types of data 

With the increasing popularity of big data and the relatively common use of technology-enhanced 
learning, some have included these new types of data in DBR studies. For instance, complex statistical 
modeling has recently taken the traditional place of qualitative approaches (Markauskaite, 2010; 
Markauskaite & Reimann, 2008), arguing this approach avoids selection and confirmation bias. 

Others remain skeptical about finding usable evidence of learning from big data, citing examples of 
contextual, interactional “in-room” events that are not logged automatically (Stevens, 2013); such events 
may explain successes and failures of designs in important ways. As an example, a long period of activity on 
a logfile might indicate a range of activities: a student spending a long time diligently reading the screen; a 
student absent from the activity, wandering the class out of boredom; or a teacher interaction in response to a 
reflective question by the student. These tell us very different things about how the design is or is not 
supporting learning, and do not, on average, provide useful design information. 

To deal with this issue, others rely on a combination of video and logfiles. For instance, researchers 
first analyzed classroom and video data to redesign a feedback feature that students rarely used (Segedy, 
Kinnebrew, & Biswas, 2012). They then analyzed data from students’ interactions with the technology using 
hidden Markov Modeling, to evaluate the impact of their design decisions, leading to further refinement of 
both the design and theory guiding their work. Similarly, in our research, we have leveraged data from 
logfiles, field notes, student performance, and videos of implementations to test and inform design decisions 
(Svihla & Linn, 2012a, 2012b). For instance, based on review of video and logfiles and student performance, 
we chose to add a step to an instructional unit to support students to interpret interactive visualizations, but 
we feared students might use a guess-and-check approach as a result. By examining logfiles, we found that 
most students revisited an earlier step seeking information, rather than guessing. This led us to more closely 
examine logfiles for particular patterns of activities, such as revisiting steps from earlier activities. Though 
the primary theory guiding that work was well developed, the instantiation of it in the particular context and 
for the particular curricular goals was not, resulting in a much more humble, localized version that 
incorporated new ideas about how students revisit prior curricula to support their learning. 


Keypoints 

■ Design-Based Research (DBR) is the core methodology of the learning sciences 

9 The purpose of DBR is to develop designs for learning and learning theory through iterative 
refinement and retrospective analysis 

S Typically, DBR involves qualitative data; recently, some researchers have begun using “big data” 
to make design refinements and build theory 

Design-Based Implementation Research (DBIR) is a recent trend that involves scaling DBR to 
support change in larger systems, such as school districts 
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